We should avoid duplicating implementations. Could we not hoist this into a separate file and import from both?

We should avoid duplicating implementations.

Agreed

Could we not hoist this into a separate file and import from both?

No 😭

In the words of Cursor Opus 4.6 1M Thinking:

There's no reasonable way to share code between the two build_hooks.py files:

• Each is a PEP 517 build backend loaded via backend-path=["."] from its own package directory (cuda_bindings/ or cuda_core/). They can't import from the monorepo root or from each other.
• The function exists because cuda.pathfinder can't be imported normally in this context — so it can't live inside cuda-pathfinder either.
• A shared helper package in build-system.requires would be massive overkill for ~15 lines of stable code.

The only alternatives are worse: a new PyPI package just for this, importlib file-path hacks, or copying a shared file into both directories at CI time.

The "please keep in sync" comments are the pragmatic solution here.

We have other existing cases of "please keep in sync" that are impractical to avoid.

It just crossed my mind: a pre-commit check to enforce that they stay in sync is probably pretty easy to implement.

Should we include some message explaining that we iterated through sys.path looking for pathfinder and couldn't find it? That seems to be important information that would be missing from just throwing we couldn't import pathfinder module.

Done: commit 3d422de

Path hacking of any sort like this is a code smell. At the very least the docstring is missing a concrete example of why this is needed.

isolated build environments

is doing a ton of work there. Isolated how? What is backend-path and where does that come from? Where are all these nouns like backend-path and build-env coming from?

In what real-world scenarios does this actually happen where there's not a viable alternative?

I'm really skeptical that path hacking like this is necessary.

It's fully documented now here: #1824 (that in turn is pointing to #1803, where you can see the original development history for the workaround)

The comment here was updated:

commit 6892ae0

This except ModuleNotFoundError is too broad: it will also catch ModuleNotFoundErrors raised from inside cuda.pathfinder itself. If cuda.pathfinder is installed but one of its own imports is broken, we silently reinterpret that as the namespace-shadowing case and take the workaround path.

I'd narrow the fallback to the exact failure we expect here:

try:
    from cuda.pathfinder import get_cuda_path_or_home
    return get_cuda_path_or_home
except ModuleNotFoundError as exc:
    if exc.name != "cuda.pathfinder":
        raise

That preserves the real traceback for genuine cuda.pathfinder import bugs and keeps the workaround limited to the cuda.pathfinder-not-found case.

Thanks for catching this (no pun intended)!

Done: commit b779d0d

Mutating cuda.__path__ here permanently changes namespace resolution for the rest of the backend process, and it selects whichever cuda/pathfinder directory happens to appear first on sys.path. That's pretty fragile.

If we keep this overall approach, I think the safer version is to resolve the installed cuda-pathfinder distribution explicitly and append only that exact cuda/ directory, instead of scanning sys.path:

from importlib import metadata
from pathlib import Path

# after narrowing the ModuleNotFoundError as above
import cuda

dist = metadata.distribution("cuda-pathfinder")
site_cuda = str(dist.locate_file(Path("cuda")))
cuda_paths = list(cuda.__path__)
if site_cuda not in cuda_paths:
    cuda.__path__ = cuda_paths + [site_cuda]

from cuda.pathfinder import get_cuda_path_or_home
return get_cuda_path_or_home

That still uses the namespace workaround, but it removes the sys.path guesswork.

If you'd rather avoid cuda.__path__ mutation entirely, another option is to load cuda/pathfinder/_utils/env_vars.py directly from the installed distribution under a private module name and call get_cuda_path_or_home() from there. That file is stdlib-only today, so it avoids rewriting namespace-package state.

I think the safer version is to resolve the installed cuda-pathfinder distribution explicitly

Yes, much better, thanks!

Done: commit 6facaa5

another option is to load cuda/pathfinder/_utils/env_vars.py directly

The answer to this goes deeper, I rewrote the PR description to make this clearer. Quoting from there:

The main point of this PR is to firmly establish cuda-pathfinder as a tool usable at build time, not just at runtime.

We don't want to advertise a helper function that reaches into pathfinder internals.

Unfortunately it looks like we have to backtrack to the sys.path scan. Here is a Cursor writeup, based on information generated via commit 3cb0313:

We tried replacing the sys.path scan with importlib.metadata per your suggestion. It works locally but fails in CI (cibuildwheel / manylinux). Here's why.

Diagnostic output (linux-64, py3.12)

dist._path: /project/cuda_bindings/cuda_bindings.egg-info
dist._path.parent: /project/cuda_bindings
locate_file('cuda'): /project/cuda_bindings/cuda
locate_file exists: True
locate_file/pathfinder exists: False
cuda.__path__ (before): _NamespacePath(['/project/cuda_bindings/cuda'])
sys.path:
  /tmp/build-env-_bna8wjb/lib/python3.12/site-packages  ->  cuda/pathfinder exists: True
cuda.__path__ (after): _NamespacePath(['/project/cuda_bindings/cuda'])

What happens

1. distribution("cuda-pathfinder") finds the project's own cuda_bindings.egg-info (created by the setuptools metadata step), not the cuda-pathfinder package installed in the build env under /tmp/build-env-…/lib/python3.12/site-packages/.

2. So dist.locate_file(Path("cuda")) returns /project/cuda_bindings/cuda — the shadowed directory itself. Adding it to cuda.__path__ is a no-op.

3. Separately, cuda.__path__ remained a _NamespacePath after our plain-list assignment. The _NamespacePath descriptor recalculates on access, discarding the mutation. (The old working code assigned cuda.__path__ = list(cuda.__path__) + [sp_cuda] without a preceding import cuda that could re-trigger recalculation.)

Conclusion

importlib.metadata is unreliable here because the in-tree egg-info shadows the installed distribution. The sys.path scan is the correct approach — it directly verifies that cuda/pathfinder/ exists on disk and is not fooled by metadata from the wrong package.

CI run with diagnostics: https://github.com/NVIDIA/cuda-python/actions/runs/23809723643

Another Cursor writeup, based on the diagnostic results:

Regarding the two concerns about the sys.path scan:

"First hit wins" ordering: In this context there is exactly one cuda-pathfinder installed in the build environment — pip deduplicates. So there is exactly one sys.path entry with cuda/pathfinder/ on disk. "First hit" finds the only hit.

Sticky cuda.__path__ mutation: The build backend is an ephemeral subprocess that builds one wheel and exits. The mutation corrects an already-wrong _NamespacePath (which only points to the project's cuda/ directory) to what it should have been (both the project's and site-packages' cuda/ directories). "Permanent" means the remaining ~seconds of that subprocess's lifetime.

And as shown in the diagnostic run, importlib.metadata cannot be used as an alternative because the in-tree cuda_bindings.egg-info shadows the installed cuda-pathfinder distribution — distribution("cuda-pathfinder") resolves to the project's own metadata, not the build-env's site-packages copy.